Papers with image embedding
Soft Knowledge Prompt: Help External Knowledge Become a Better Teacher to Instruct LLM in Knowledge-based VQA (2024.acl-long)
Copied to clipboard
| Challenge: | Recent research focuses on improving prediction performance and reliability of LLM. |
| Approach: | They propose a method to actively extract valuable information from the knowledge to produce a latent vector as a soft prompt, which is fused with the image embedding to form a knowledge-enhanced context to instruct LLM. |
| Outcome: | The proposed method improves performance on knowledge-based VQA benchmarks. |
Bridging Semantic and Modality Gaps in Zero-Shot Captioning via Retrieval from Synthetic Data (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for zero-shot image captioning fail to bridge the semantic gap between images and textual inputs. |
| Approach: | They propose a retrieval-based framework that leverages only existing synthetic image-text pairs as its search corpus to bridge the gap when using synthetic data for captioning. |
| Outcome: | The proposed method bridges the semantic gap between a synthetic image and its input text . it extracts image-related textual descriptions to mitigate the modality gap during decoding . |
"Penny Wise, Pixel Foolish": Bypassing Price Constraints in Multimodal Agents via Visual Adversarial Perturbations (2026.findings-acl)
Copied to clipboard
| Challenge: | Mobile Agents are a key component of the “Agentic Economy” where they perform high-stakes financial transactions. |
| Approach: | They propose a systemic vulnerability termed Visual Dominance Hallucination (VDH) VDH exploits the modality gap in CLIP-based encoders via a novel Semantic-Decoupling Loss. |
| Outcome: | The proposed framework exploits the modality gap in CLIP-based encoders by preserving fidelity. |